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ABSTRACT 

This  position  paper  describes  a  vision  of  how  Chemical 
Biological  Radiological  and  Nuclear  (CBRN)  situation  awareness 
and  threat  assessment  can  benefit  from  the  use  of  Gesture- 
Directed  Sensor-Information  Fusion  (GDSIF).  It  describes  a 
concept  of  operations  for  the  war  fighter’s  use  of  electronic 
wireless-communication  gloves  (eGloves)  to  communicate,  plan, 
and  react  while  wearing  Mission-Oriented  Protective  Posture 
(MOPP)  gear.  It  also  suggests  the  benefit  of  using  the  hardware 
and  firmware  resident  on  the  eGloves  to  perform  data  fusion  from 
environmental  sensors.  It  provides  a  roadmap  of  research  issues 
and  challenges  that  will  need  to  be  overcome  to  realize  this 
technological  advance,  and  provide  inspiration  for  other  problem 
domains. 

Keywords:  chemical,  biological,  radiological,  and  nuclear 
warfare;  electronic  data  glove;  gesture-signal  communications; 
sensor-information  fusion;  situational  awareness;  human  factors; 
design  reliability. 

1.  INTRODUCTION 

New  modalities  of  computer  interface  are  a  prominent  theme  of 
the  past  few  years.  Touchscreen  interfaces  have  exploded  in 
popularity,  especially  with  the  iPhone  and  Android  mobile 
operating  systems.  ‘Touchless’  interfaces  are  also  gaining 
prominence.  Just  within  recreation,  the  Nintendo  Wii,  Playstation 
Move,  and  Microsoft’s  Project  Natal  have  recently  come  to  the 
public’s  consciousness,  fulfilling  years  of  promise  and  research. 
This  paper  will  describe  a  vision  of  how  these  types  of 
technologies  can  be  pushed  even  further,  incorporating  language, 
intent,  and  function  beyond  just  a  simple  interface  device.  We 
hope  this  case  study  will  provide  inspiration  for  other  problem 
domains. 

The  battle  space  is  full  of  threats  that  would  be  better  to  detect 
earlier  rather  than  later.  War  fighters  are  in  the  best  position  to 
detect  CBRN  threats  in  theatre.  Current  Chemical-protection  for 
war  fighters  on  the  ground  inhibit  electronic  communication  via 
keyboards,  cell  phones,  and  remote-control  devices.  War  fighters 
need  better  communication  methods  in  hazardous  environments 
characterized  by  CBRN  agents.  Important  topics  in  wartime 
communications  include  but  are  not  limited  to  situation  and  threat 
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assessments.  To  improve  communications  capabilities  for  the  war 
fighter  wearing  protective  gear  in  hazardous  environments,  a 
series  of  eGloves  have  been  developed  with  a  view  toward  freeing 
the  war  fighter  of  the  need  to  type  on  a  keyboard  while  wearing  a 
MOPP  suit.  (See,  for  example,  [1]  and  [2].)  These  eGloves  can 
help  the  war  fighter  transmit  gestures  with  the  hands  and  fingers 
from  within  the  protective  gear  [1],  [2],  or  they  can  be  used  to 
transmit  encoded  ASCII  characters  [1].  Fig.  1  shows  an  eGlove 
prototype. 

Research  has  demonstrated  that  chemical  and  biological  sensors 
can  be  made  smaller,  more  sensitive,  more  species  specific,  and 
easier  to  deploy  in  a  variety  of  ways.  (See,  for  example,  [6]  and 
[7].)  The  current  eGloves  have  magnetic  and  motion  sensors  for 
gesture  recognition  [1],  [2],  An  important  future  step  to  enhance 
the  effectiveness  of  the  war  fighter  is  to  integrate  CBRN  and  other 
sensors  into  the  eGloves.  In  addition  to  simply  recording  CBRN 
environmental  data,  development  of  efficient  data-fusion 
algorithms  to  fuse  the  information  from  CBRN  and  other  sensors 
with  gestures  from  the  eGlove  operator  will  provide  situational 
awareness  beyond  the  readings  from  individual  sensors. 


Figure  1.  eGlove  with  motion  sensors  and  CPU  circuitry 
(Patents  pending  12/323,986;  12/325,046) 

The  following  sensors  could  be  integrated  with  gestures 
transmitted  using  the  eGlove:  CBRN  sensors,  acoustic  sensors, 
Geo-Positional  Sensors  (GPS),  optical  sensors,  physiological 
sensors  to  monitor  the  health  of  the  operator,  and  imagery  sensory 
input  from  video  cameras  mounted  in  various  strategic  locations 
in  the  environment.  The  output  of  these  passive  sensors  can  be 
transmitted  with  the  magnetic/motion  sensor  output.  This  will 
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improve  communications  by  providing  explicit  measurements  or 
infonnation  without  burdensome  and  error  prone  manual  relay. 
Integrating  the  environment  sensor  data  with  the  current  eGlove 
gesture  software  would  expedite  communications  by  providing 
contextual  information  to  the  glove.  For  example,  if  the 
physiological  sensors  on  the  eGlove  are  detecting  values  beyond 
normal  thresholds,  different  communications  suggestions  could  be 
presented  than  during  normal  operating  conditions. 

The  improved  hardware  alone  provides  tremendous  value, 
especially  in  the  extreme  CBRN  conditions.  However,  this  paper 
will  focus  on  one  bridge  further,  a  proposed  interaction  paradigm, 
rather  than  the  interface  technology. 

2.  DESCRIPTION  &  BENEFIT 

Gesture-Based  Sensor-Information  Fusion  (GBSIF)  refers  to  the 
fusing  of  sensor  data  collected  from  the  environment  with  data 
from  motion  sensors  on  the  eGlove.  The  eGlove  features  a  Central 
Processing  Unit  (CPU)  that  is  used  to  fuse  hand  and  finger 
motions  and  positions  into  gestures,  as  shown  in  Fig.  1.  The  same 
CPU  can  be  used  to  fuse  additional  data  from  the  environment.  In 
GBSIF,  the  operator  transports  the  sensor  array  but  does  not  take 
an  active  role  in  determining  the  sensors  that  will  participate  in 
the  fusion  or  the  target  subjects  about  which  the  data  will  be 
collected,  with  the  exception  of  the  sensors  mounted  on  the 
eGlove.  Data  are  collected  from  the  environment  and  also  from 
the  glove  sensors,  and  these  data  can  be  fused  and  integrated  on  a 
network  site  that  differs  from  the  user’s  node.  Thus,  gesture 
sensor  data  and  environmental  data  are  collected,  fused,  and 
integrated  where  appropriate.  However,  the  gestures  themselves 
are  not  the  primary  driving  force  in  selecting  information  sources 
and  controlling  the  fusion  process. 

GBSIF  is  passive  from  the  point  of  view  of  the  operator.  They  are 
serving  to  extend  the  battlespace  sensor  network  in  two  very 
important  and  valuable  ways:  providing  mobile  environmental 
sensors  that  serve  to  augment  everyone’s  situational  awareness, 
and  providing  a  new  type  of  sensor  reporting,  gestures.  For 
example,  consider  a  simple  eGlove  instrumented  with  a  chemical 
sensor  and  a  GPS  sensor  in  addition  to  the  standard  eGlove 
magnetic  and  motion  gesture  recognition  sensors.  War  fighters 
progressing  through  a  battlespace  are  now  providing  additional 
sensor  information.  This  is  additional  data  is  not  intended  to 
replace,  but  to  augment  to  unattended,  remote  operated,  and 
autonomous  sensors  or  robots.  [3][4] 

For  example,  some  war  fighters  might  be  in  close  proximity  to  a 
chemical,  so  the  combination  of  reports  from  all  their  sensors 
(some  sensors  reporting  a  negligible  value,  some  reporting  a  mild 
value,  some  reporting  a  moderate  value)  in  conjunction  with  the 
GPS  sensors  would  provide  a  general  idea  of  the  location  and 
concentration  of  a  potential  chemical  plume.  Continuing  the 
example,  sensors  recorded  and  transmitted  some  of  the  war 
fighters  were  gesturing  with  ‘warning’  or  ‘stop’  gestures.  GBSIF 
allows  for  the  chemical,  GPS,  and  gestures  to  be  fused  providing 
new  situational  awareness,  in  this  example  the  gestures  would 
provide  more  information  about  the  origin  of  the  chemical 
detection. 

In  contrast,  Gesture-Directed  Sensor-Information  Fusion  (GDSIF) 
includes  GBSIF  but  extends  it  to  include  the  active  participation 
of  the  eGlove  operator  to  initiate  sensor-information  fusion.  The 
concept  of  operation  of  the  GDSIF  is  that  the  war  fighter  would 
point  to  or  otherwise  select  a  platform  or  another  object  in  the 
battle  space  using  a  gesture  while  wearing  a  GDSIF-equipped 
eGlove,  including  sensors  onboard  the  eGlove  itself.  The  eGlove 
would  be  linked  to  reference  sensors  to  determine  orientation  and 


azimuth  of  the  operator’s  ann.  The  eGlove  also  would  use  GPS  to 
detennine  the  operator’s  geographic  location.  Gestures  would  cue 
sensors  to  send  their  data  to  the  eGlove  where  these  data  would  be 
fused  with  the  gesture  that  prompted  the  data  collection.  Fusion 
would  be  accomplished  in  the  CPU  mounted  on  the  eGlove. 

The  advantage  of  GDSIF  over  GBSIF  is  clear.  The  war  fighter, 
with  immediate  access  to  the  environment  and  data,  can  start  to 
process  and  add  value  to  the  infonnation  being  collected.  Ideally, 
it  will  not  only  result  in  added  value,  but  if  the  gestures  are 
efficient,  it  can  be  faster  as  well. 

The  distinction  of  where  computation  is  taking  place  is  key, 
because  GBSIF  fusion  is  accomplished  remotely,  and  can  use  the 
virtually  unlimited  resources  available  in  the  cloud.  In  order  for 
the  infonnation  to  be  available  in  a  timely  manner,  given  cunent 
battlefield  bandwith  constraints,  GDSIF  algorithms  need  to  be 
able  to  be  executed  with  the  resources  available  to  the  operator. 

In  a  scenario  similar  to  the  one  described  above,  with  a  number  of 
distributed  war  fighters  equipped  with  eGloves  with  chemical 
sensors,  and  one  individual  war  fighter  wants  to  know  what 
direction  to  proceed  to  locate  and  eliminate  the  source  of  the 
chemical.  He  is  wearing  heavy  MOPP  gear  to  protect  against  the 
chemical,  but  this  also  limits  his  communication  ability.  By 
sending  a  gesture  to  nearby  sensors,  the  war  fighter  can  receive 
feedback  as  to  whether  a  particular  direction  would  have  a  higher 
or  lower  concentration  of  the  chemical  than  his  current  location. 
The  fusion  algorithm  would  be  keyed  by  the  war  fighter’s  initial 
gesture.  The  infonnation  is  fused  from  his  own  sensor,  and  the 
results  of  querying  nearby  sensors.  An  algorithm  determines 
relative  concentrations,  and  provides  the  result  the  war  fighter 
desires. 

More  complicated  fusion  algorithms,  in  regards  to  correlation  of 
multiple  different  types  of  sensors,  as  well  as  just  higher  levels  of 
complexity  [5],  constitute  their  own  area  of  research.  GBSIF  will 
incorporate  advances  in  generalized  data  fusion,  but  is  itself  a 
more  specialized  area  of  research. 

Sensors  added  which  can  detect  imagery  can  add  great  meaning  to 
the  user's  experience.  Multiple  sensors  which  can  fuse  imagery 
can  provide  a  description  of  what  the  war  fighter  sees.  Integration 
of  visual  imagery  with  gesture  can  yield  much  insight  into  the 
cognitive  experience  of  the  user.  For  example,  if  the  user 
provides  a  gesture  that  represents  alarm,  he/ she  might  not  be  able 
to  have  enough  time  to  indicate  what  the  alarm  is,  but  with  the 
fusion  of  the  gesture  and  the  imagery  data,  this  can  be  more  easily 
inferred.  Further,  with  sensors  providing  imagery  data,  there 
might  be  too  much  infonnation  being  sent  to  the  cloud  for 
accurate  processing.  However,  if  the  user  of  can  provide  a  start 
and  stop  mechanism,  the  linking  between  the  gesture  and  the 
corresponding  imagery  can  be  made,  giving  more  insight  into  the 
cognitive  experience. 

If  the  war  fighter  is  given  two  gloves,  then  by  using  both  hands,  it 
is  possible  to  select  a  region  to  be  further  investigated.  Imagine 
looking  across  the  street  and  then  selecting  a  region  of  a  building 
with  two  hands.  If  the  location  of  the  imagery  data  is  known, 
than  mathematical  processing  can  be  done  to  detennine  a  specific 
region  by  simply  holding  two  hands  in  the  air.  Similarly,  with  the 
thumb  and  index  finger  one  can  make  a  circular  shape  which  can 
similarly  be  used  to  detect  a  specific  region.  Given  the  angle  of 
the  thumb  and  index  finger,  one  can  than  detennine  the 
circumference  of  the  circle  made  by  these  two  fingers,  and  then 
locate  the  respective  region  on  the  sensor  imagery  data.  These 
gestures  are  not  unlike  some  cunent  gesture  systems,  such  as 
SixthSense[10],  however,  use  of  a  camera  for  gesture  recognition 


is  absolutely  not  an  option  for  a  CBRN  war  fighter  in  a  battlefield 
scenario. 

Multi  sensor  fusion  is  commonly  used  for  identifying  specific 
regions.  However,  there  is  always  a  probability  associated  with 
this  detection.  Feature  vectors  comprise  kinematics  parameters 
such  as  position  vector,  velocity,  vector,  and  acceleration  vector 
[9].  These  features  are  used  to  help  determine  the  location  of  the 
specific  regions.  Estimated  fused  target  probabilities  are  used  to 
help  predict  which  object  is  being  recognized.  Integrating  a  glove 
with  sensory  image  data  could  improve  these  probabilities. 

3.  PROPOSED  ARCHITECTURE 

Preliminary  experiments  show  that  the  current  CPU  will  support 
additional  sensors  without  difficulty.  And  new  low-power  CPU 
technology  is  available  on  a  near-daily  basis,  so  this  will  continue 
to  be  less  of  an  issue.  Of  the  many  options  for  accomplishing  this, 
the  three  subsequent  architectures  provide  the  best  advantages. 

1.  Tracking  the  date-time  group  of  the  sensor  data,  the  GPS 
geographic  location,  and  the  sensor  type  will  provide  the  pedigree 
metadata  [8]  necessary  to  sort  and  fuse  the  information.  CBRN, 
acoustic  and  GPS  sensors  can  be  mounted  on  the  eGlove  and  the 
CPU  can  collect  and  fuse  data  from  all  sources  on  the  eGlove. 
The  CBRN,  acoustic  and  GPS  sensors  would  detect 
environmental  conditions  but  not  health-related  conditions  of  the 
operator.  Redundant  sensors  mounted  on  the  eGlove  could  help 
reduce  problems  due  to  false  alanns  of  any  one  sensor.  CBRN 
Sensors  with  heightened  sensitivity  could  transmit  detection  even 
before  the  war  fighter  is  aware  of  the  presence  of  a  chemical  or 
biological  agent.  These  networked  and  fused  sensors  could 
provide  warnings  and  alerts  to  be  sent  not  only  to  the  operator  but 
also  to  the  wireless  network  with  which  the  operator  is  in 
communications.  When  applicable,  these  alerts  could  be  localized 
to  prevent  ‘noise’  being  sent  to  unaffected  parties.  This  could  be 
used  to  alert  and  send  early  emergency  response  team 
independent  of  any  specific  request  from  the  war  fighter. 

2.  This  architecture  is  the  same  as  the  one  described  in  item  1,  but 
in  addition  to  external  sensors,  the  eGlove  also  would  detect 
physiological  data  of  the  operator,  such  as  pulse  and  Galvanic 
skin  response  (GSR).  For  example,  if  the  chemical  detectors  sent 
signals  about  a  plume  and  the  operator  did  not  transmit  any 
gestures  determined  to  have  intent  or  meaning,  this  could  mean 
that  the  chemical  agent  had  attacked  and  incapacitated  the 
operator.  The  pulse  and  other  physiological  signals  could  help 
determine  the  operator’s  health. 

3.  The  CPU  mounted  on  the  eGlove  could  be  programmed  to 
detect  not  only  local  signals  from  hand  and  finger  gestures  but 
also  from  a  variety  of  wireless  sensors  throughout  the  battle 
space.  This  could  include  sensors  mounted  on  or  deployed  from 
UAVs  [7],  motor  vehicles,  or  even  other  war  fighters.  This 
provides  a  wider  coverage  of  the  battle  space  beyond  the 
immediate  vicinity  of  the  operator.  The  obvious  disadvantage  of 
these  extra  sensors  is  that  it  adds  complexity  to  the  task  of  the 
CPU  on  the  eGlove.  The  challenge  is  to  develop  effective  data- 
fusion  algorithms  to  manage  the  data  streams  and  the  metadata 
from  an  increased  number  and  variety  of  sensors  in  the  battle 
space. 

The  operator  could  select  a  1 )  “Raw-data-only  mode”  to  transmit 
the  data  to  the  wireless  network,  or  2)  “fuse  data”  mode  in  which 
algorithms  stored  in  the  eGlove’s  CPU  would  perfonn  data  fusion 
and  then  transmit  the  fused-data  result  to  the  wireless  network. 
The  operator  could  use  gestures  to  control  multiple  modes  of 
operation  of  the  eGlove. 


4.  GESTURES 

Simple  gestures  can  be  used  to  communicate  information  to 
improve  situational  awareness,  send  commands  to  personnel  and 
to  robots  [2],  and  send  commands  to  CBRN  and  other  sensors  in 
the  battle  space.  For  example,  one  potential  information  fusion 
would  be  to  point  at  a  sensor-data  source  in  the  battle  space  with 
the  index  finger  extended  and  the  other  fingers  touching  the  palm, 
(to  distinguish  it  from  similar  gestures  that  use  the  whole  hand  to 
point.)  This  pointing  gesture,  when  recognized,  would  signal  the 
sensor  and  trigger  a  data  stream  or  a  single  reading  from  the 
designated  sensor  to  the  local  common-data  backbone. 

Successful  transmission  from  the  sensor  would  trigger  haptic 
feedback  [1]  on  the  operator’s  glove  indicating  that  the  data  set 
has  been  sent  to  the  network.  Continuing  the  example,  the  war 
fighter  could  repeat  the  pointing  process  with  a  second  sensor  and 
then  a  second  gesture,  for  example  a  fist  with  the  ann  held 
straight  down,  would  trigger  a  pre-detennined  sensor-information 
fusion  process.  Using  the  fusion-fist  gesture  in  this  manner  would 
distinguish  it  other  gestures  that  employ  a  closed  fist  with  the  ann 
extended,  which  in  some  command  contexts  means  “stop.”  It  also 
would  avoid  confusion  with  gestures  in  which  the  fist  is  held 
close  to  the  chest. 

CBRN  sensor-infonnation  could  be  thus  queried  and  fused  with 
each  other  and  also  with  information  from  other  sources  in  the 
battle  space,  such  as  electromagnetic  or  acoustic  sensors.  In  more 
advanced  implementations  of  this  methodology,  the  war  fighter 
could  provide  some  degree  of  input  about  what  type  of 
information  is  desired  from  the  sensor-infonnation  fusion,  such  as 
a  prediction  of  routes  to  use  for  relatively  safe  travel  with  respect 
to  the  deployment  of  CBRN  agents. 

Specific  additional  developments  would  likely  arise  organically 
from  close  collaboration  with  the  war  fighters  using  the  tool, 
following  an  approach  similar  to  the  ‘User  Centered  Design’ 
software  approach. 

5.  DIRECTIONS  FOR  FUTURE  RESEARCH 

Further  research  is  needed  in  this  area.  To  facilitate  GBSIF  and 
GDSIF,  the  vocabulary,  syntax,  and  semantics  of  command 
gestures  would  need  to  be  developed  to  include  all  possible  modes 
of  operation  that  the  war  fighter  would  require.  Because  the 
eGlove  is  designed  for  use  in  a  variety  of  situations,  the 
vocabulary  of  gestures  for  GDSIF  ideally  would  build  on  the 
vocabulary  of  gestures  already  in  use  for  similar  purposes.  For 
example,  special  forces,  such  as  Navy  Sea  Air  and  Land  (SEAL) 
teams  and  Army  Rangers,  use  gesture-based  communication 
designed  for  covert  operations  where  stealth  and  silence  are 
requirements.  Human  factors  and  physiology  would  help 
detennine  the  ease  with  which  some  gestures  rather  than  others 
could  be  used.  Special  fusion  algorithms  would  process  gesture 
data  and  fuse  them  with  environmental  and  physiological  data. 
Advanced  operators  could  be  trained  to  issue  gesture  commands 
to  determine  which  fusion  algorithms  to  execute. 

Maintaining  the  integrity  and  timeliness  of  the  Common 
Operating  Picture  (COP)  with  existing  information  is  already  a 
challenge  [5],  Integrating  GDSIF  infonnation  into  the  COP  where 
other  war  fighters  could  benefit  from  local  observations  is  an  even 
greater  challenge  requiring  further  research  and  testing.  Potential 
applications  of  the  GDSIF  can  be  integrated  into  the  Joint 
Warning  and  Reporting  Network  (JWARN)  program  of  record. 
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